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ABSTRACT 

The symposium to which the present paper is an 
introduction presents results from a 1970 survey in which over 12,000 
high school juniors were tested using selected portions of the 1960 
Project TALENT battery. This paper discusses the methodology of the 
1970 survey. It also presents statistics which summarize the 
performance of the current eleventh grade sample on the TALENT 
reading comprehension scale. These statistics indicate that little if 
any gain has been recorded ir: reading comprehension ability over the 
past decade. Such a finding appears to deserve considerable weight in 
suggesting directions for educational improvement efforts. (Author) 
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PROGRESS IN EDUCATION 1960-1970: 
A SAMPLE SURVEY 



Steven M. dung 

American Institutes for Research 
Palo A1 to, Cal ifornia 



ABSTRACT 



The symposium to which the present paper is an introduction presents 
results from a 1970 survey in which over 12,000 high school juniors were 
tested using selected portions of the 1960 Project TALENT battery. This 
paper discusses the methodology of the 1970 survey. It also presents 
statistics which summarize the performance of the current eleventh grade 
sample on the TALENT reading comprehension scale. These statistics indicate 
that little if any gain has been recorded in reading comprehension ability 
over the past decade. Such a finding appears to deserve considerable 
weight in suggesting directions for educational improvement efforts. 




2 



INTRODUCTION 



The Project TALENT survey (Flanagan et al , 1962; Flanagan, 1969) 
has provided some of the best data available regarding the characteristics 
and deficiencies of secondary educational programs in the United States. 

TALENT is a large-scale psychometric investigation which began with the 
testing of over 440,000 secondary pupils in the spring of 1960. One year 
and five year follow-up studies have now been completed on samples of students 
from this group who took the battery as high school seniors and juniors. 

These follow-up data provide Valuable insights into t.he relationships between 
various measures of ability and interest obtained in high school and later 
career development (Quirk, 1969; TALFNT Staff, 1971). 

Early in 1S70 Dr. John Flanagan, Director of Project TALENT, proposed 
the testing of a small representative sample of current high school juniors 
in order to estimate changes which have occured during the decade since 
the original TALENT survey. With the aid of general support funds provided 
by the National Institutes for Health, this operation, called Progress 
in Education Survey (PIES), was conducted by the Palo Alto Office of the 
American Institutes for Research from April through June of 1970. A strati- 
fied random sample of schools was selected from the population of schools 
which participated in the 1960 TALENT survey. In all, 12,722 11th graders 
were tested in 134 schools around the country. The preliminary results 
are reported in this paper and in the following one by Dr. Flanagan. 
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METHOD 



The PIES Battery 

The survey instrument consisted of a 48 item multiple-choice reading 
comprehension scale, 40 student information items, and a single essay item. 

It was designed to be administered within the time confines of a 55 minute 
school period, with the reading scale requiring 30 minutes and the remainder 
untimed but requiring approximately 20 minutes. 

The reading comprehension scale was identical to the scale contained 
in the 1960 TALENT battery. It consisted of eight passages which were to 
be read, with each passage followed by several 5 option multiple-choice 
questions testing comprehension of the passage rather than mere ability 
to recognize words. Passages dealt with social studies, natural science, 
and literary content. Poetry as well as prose passages were included. In 

answering the questions, students were permitted to refer back to the passages. 
Extremely difficult vocabulary was avoided; in general, the words were within 
the first 15,000 entries on the Thorndike-Lorge List (1944). It was felt 
that reading comprehension was a particularly good ability to tap, in view 
of the effectiveness of reading as a predictor of general school success 
and In view of the attention that has been paid to Improving reading skills 
In educational programs of the past decade. 

Thirty-six of the 40 student Information items were adapted from the 
1960 TALENT Student Informatlor. Blank; the essay question, which asked students 
to write a short paragraph on their views about an Ideal occupation, had 
also been Included in the 1960 battery. In addition, a questionnaire was 
completed by the principal of each participating school. This questionnaire 
attempted to gather Information about changes which have taken place In 
O neighborhoods and curricula of these schools since I960. 
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PIE Administration 

Sample . The primary goal of the PIES sampling procedure w'. to secure 
a representative and accessable group of high school juniors within the 
budget and time constraints which prevailed. Thus, instead of selecting an 
entirely new sample, it was decided to utilize the 1960 Project TALENT 
sample. Although this meant some problems in terms of school age, to be 
mentioned later, it insured that a randomly selected pool of schools, already 
stratified on the basis of type of school {public vocational, public 
non-vocational , private, or parochial), geographical region, and size, 
could be readily identified. Furthermore, due to the previous Project 
TALENT contacts with these schools, there was a higher probability of securing 
participation on the relatively short notice that was available. Readers 
interested in examining in detail the characteristics of the TALENT sample 
are referred to previous publications devoted to that topic (e.g., Flanagan 
et al , 1962, pp. 43-56). 

It was decided that a twenty percent random sample from the TALENT 
pool was sufficient for the purposes of the 1970 survey. Thus one fifth 
of the secondary schools which participated in th? 1960 TALENT survey 
(199 out of 989) were randomly selected to compose the PIES sample. Table 1 
presents the sunmary statistics for this sample for each of the TALENT 
strata. The 1970 weight, which appears in the right column of this Table, 
represents the factor by which the scores of each particpating school in 
that stratum were Increased in order to reproduce the characteristics of 
the original stratum. These weights were later combined with the 1960 
TALENT weights to estimate the characteristics of the population from which 
the 1960 sample was drawn. Appendix A contains an explanation of each 
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Insert Table 1 hare 



Securing participation . A personal letter was sent to the school 
superintendent of each district which contained a selected school requesting 
cooperation in administering the PIES battery. A copy was also sent to the 
principal of the school. Two weeks later members of the AIR project staff 
called each principal, again requesting his or her cooperation. This 
procedure revealed that a considerable number of schools which had tested 
in 1960 no longer existed. In cases where a consolidation had ocurred, 
the principal of the consolidated school was contacted and his cooperation 
solicited. In all, 25 of the 199 schools were no longer opc.uting. Most, 
of these were parochial schools or small public schools in the rural South. 

Of the remaining 174, 148 or 85% agreed to test. Later problems with mail 
service or within schools caused 14 additional schools to be unable to 
complete testing before summer vacation. This left an N of 134 schools, 
or 77%, in the final PIES sample. Table 2 presents a sumr-ry of these figures 



Insert Table 2 here 



Administration procedure s. Participating schools were urged to ad- 
minister the survey booklet to all juniors. However, in thirty cases, 
notably ir New York City and Chicago, it was necessary to select a ten or 
twenty percent random sample of students. Tests were sent to the principal, 
or his designated subordinate, for coordination; testing then occured in 
a group administration or in classes in which all juniors were enrolled, 
such as English or homeroom. Instructions for administration were designed 
to be read verMtim by a single administrator or classroom teacher. 
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A total of 12,722 students (6,015 boys and 6, 757 girls) took the 1970 
survey. Answer sheets and principal's questionnaires were returned via 
prepaid first class mail; booklets, etc. were returned via REA express collect. 

Processing of results . All student responses were recorded on IBM 1230 
answer sheets. When the answer sheets were received in Palo Alto they were 
recorded and scored on an IBM 1230 optical test scoring machine . Reading 
scores and Individual item responses were punched into cards as a product 
of the scoring operation. These card images were then recorded on magnetic 
computer tape from which further analyses of the data were generated. Initial 
processing operations produced alphabetical listings of all students who 
took the battery, along with their raw reading comprehension score (number 
of items answered correctly), which were returned to the participating 
schools for their use. 



RESULTS AND DISCUSSION 

Reported here are the results of the PIES Reading Comprehension scale 
In comparison with the results of the 1960 administration of the same scale. 
Results of the student information items will be reported in the following 
paper. Analyses of the principal's questionnaires and the essays on ideal 
occupation have not yet been completed. 

Table 3 presents the basic unweighted descriptive statistics of the 
raw scores on the 48 item reading scale. 

Insert Table 3 here 

Whe assistance of the Arizona State University Testing Service, which 
performed the scoring operations, is gratefully acknowledged. 
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The range of scores was from 0 to 48, with Qj =24, median = 33, and = 40. 
By applying the 1970 PIES weights to these scores in conjunction with the 
i960 TALENT weights for the PIES schools, Table 4 was generated. 

Insert Table 4 here 

This table enables a comparison to be drawn with the weighted results of 
the 1960 TALENT reading comprehension scale, presented in Table 5. 

Insert Table 5 here 

The 1970 weighted mean of 31.25 compares to the 30.81 weighted 1960 
mean. Such stability of performance over the 10 year period seemed to be 
a consequential finding. Before further interpretation was attempted, 
however, consideration was given to the hypothesis that the 1970 data 
might be biased in some unknown way as a result of the age of schools which 
participated in the study. 

Since only schools which existed in 1960 could have been selected for 
the PIES sample, it seemed possible that the significant number of new 
schools, contained a possibly different student population, would have 
Influenced the results considerably had a new sample been selected. In 
order to estimate the effect of this bias In the 1970 sample, an analysis 
was run on the 1960 TALENT reading data to separate the results stenming 
from schools which were less than and greater than 10 years old. This 
analysis, described in Appendix B, produced a correction of .02 in the 
overall 1960 mean reading comprehension score. This correction is reflected 
In Table 6. 

Insert Table 6 here 
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The comparison of 1960 data, corrected for the influence of schools 
less than 10 years old, with the 1970 data results in a net gain of less 
than one half of one item answered correctly, or less than one-twantieth 
of one standard deviation. To put this gain in another perspective, it may 
be noted that the difference between the performance of juniors and seniors 
on the 1960 administration of this reading comprehension scale was 2.47 
(12th grade mean = 33.30, s.d. = 10.14). 

On the basis of these data, it was felt that there was some justifiction 
for renaming the 1970 survey Lack-of-Progress in Education. 1960-1970. 

Although these findings viewed as a whole are somewhat discouraging, 
it is apparent that some schools have made impressive gains in reading 
comprehension performance since 1960. Twenty schools gained at least four 
raw score units since 1960. Research is currently in progress to determine 
the nature of changes which have occurred in these schools. 
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Table 1 



1970 PIE Summary Testing Statistics 
By TALENT Stratum 



TALENT stratum 
Code t 



No, of schools 
in 1960 sample 



No. of schools 
sampled in 1970 
PIE 



No. of schools 
participating 
in 1970 PIE 



10 


35 


7 


7 


*1 


26 


5 


4 


22 


55 


11 


7 


31 


5 


1 


0 


32 


21 


4 


2 


41 


4 


1 


1 


42 


47 


10 


10 


43 


31 


6 


5 


44 


22 


5 


5 


51 


24 


5 


3 


52 


45 


9 


6 


53 


42 


8 


3 


54 


101 


20 


11 


61 


12 


2 


2 


62 


83 


17 


11 


63 


138 


28 


18 


64 


132 


27 


20 


91 


114 


23 


14 


92 


52 


10 


J> 


Total 


989 


199 


134 



0 




1970 Weight 



5.0 



6.5 

7.85 



10.5 

4.0 
4.7 
6.2 

4.4 

8.0 

7.5 
14,0 

9.2 

6.0 

7.55 

7.67 

6.6 

8.15 

10.4 



11 



- 10 - 



Table 2 



1970 PIE Summary Statistics 



N % 



Schools which tested 


134 


11 


Schools which refused to test 


26 


15 


Schools which agreed to test but for 
some reason could not carry through 


14 


8 




174 


100 


Schools closed or no longer having a 
junior class 


23 




Total schools selected from 1960 
TALENT 8 ample 


199 






12 



11 



Table 3 

Unweighted 1970 PIE Reading Comprehension 



Sex 


N 


Mean 


St. Dev. 


Male 


6015 


30,84 


10.84 


Female 


6757 


32.33 


10.11 


Total 


12772 


31.63 


10.49 
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Table 4 

Weighted 19/0 PIE Reading Comprehension 



Sex 


N 


Mean 


St. Dev 


Male 


1633964 


30.80 


11.07 


Female 


1793682 


31.66 


10.53 


Total 


3427646 


31.25 


10.79 
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Table 5 

Weighted 1960 Talent Reading Comprehension 
For Juniors* 



Sex 


N 


Mean 


St. Dev, 


Male 


949666 


30.52 


10.97 


Female 


976390 


31.09 


10.23 


Total 


1926056 


30.81 


10.60 



*Not corrected to remove influence of schools less than 10 years old 
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Table 6 

Weighted 1960 Talent Reading Comprehension 
For Juniors* 



Sex 


N 


Mean 


St. Dev. 


Male 


949666 


30.32 


11.07 


Female 


976390 


31.32 


10.28 


Total 


1926056 


30.83 


10.67 


Corrected to remove 


influence 


of schools less 


than 10 years old 
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APPENDIX A 



Project TALENT School Taxonorny Code 
For Public Secondary Schools 

10. Vocational high schools 

All vocational and trade high schools 

21-64. Non-vocational high schools: (General comprehesive , academic or 

college p.eparatory, university, and superior student schools) 

21-22. Cities "A": Largest cities (1,500,000 or more) 

21. Low cost housing and low income areas 

22. Moderate and high cost housing 

31-32. Cities "B": Large cities (250,000-1,499,999) 

31. Low cost housing and low income areas 

32. Moderate and high cost housing 

41-44. Northeast: U.S.O.E. Regions 1 and 2 (Me., N. H., Vt., Mass., 

R. I., Conn., N. Y., N. J., Pa., Del., Md., D. C.) 

41. Urban (5 ,000- 249,999) - low cost housing and low income areas 

42. Urban (5,000-249,999) - moderate and high cost housing 

43. Small town 

44. Rural 

51-54. Southeast: U.S.O.E. Region 5 ( Va . , W. Va., N. C. , S. C. , Ga., 

Fla., Ky. , Tenn. , Ala., Miss., Ark., La.) 

51. Urban (5 ,000-249 ,999) - low cost housing and low income areas 

52. Urban (5,000-249,999) - moderate and high cost housing 

53. Small town 

54. Rural 

61-64. Midwest and West: U.S.O.E. Regions 3, 4, 6, 7, 8, 9 (All states 

other than those listed above) 

61. Urban (5,000-249-999) - low cost housing and low Income areas 

62. Urban (5,000-249-999) - moderate and high cost housing 

63. Small town 

64. Rural 

91. Parochial schools 

92. Private schools 
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APPENDIX B 

PROCEDURES USED TO CALCULATE 
CORRECTION FOR INFLUENCE 
OF SCHOOLS LESS THAN 10 YEARS OLD 

The procedure was complicated by the fact that less than half of the 
1960 TALENT schools reported school age on the School Characteristics Question 
naire. Tiorefore, the first step was to obtain the difference between the 
mean reading comprehension scores of 11th graders whose schools did report 
school age. Table B-l reflects these data. 



Table B-l 

1960 TALENT Reading Comprehension by Sex and School Age 



Sex 


School Age 


Weighted Mean 


Composite Mean 


Male 


< 10 yrs 


29.85 


30.31 




>_ 10 yrs 


30.51 


Female 


< 10 yrs 


31.61 


31.09 




> 10 yrs 


30.86 


Total 


<10 yrs 


30.78 


30.71 




>_ 10 yrs 


30.69 




Not reported 


30.90 






Total 


30.81 





The male, female, and total means for schools greater than 10 years 
old were subtracted from the composite means, yielding corrections of -.20 
for males, +.23 for females, and +.02 overall. 

These corrections were then applied to means for all schools which are 
contained in Table 5, yielding the results contained in Table 6. 
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